Test reporting

This document describes the test reports generated by the miniHIL.

HTML Test Report (preview)

The HTML Test Report integrates the artifacts generated during test execution and combines them with test documentation and meta information into an interactive webpage.

Generating the test report

Currently the test report can only be generated if the test cases have been executed using the runTests gradle task (e.g. .\gradlew.bat runTests).

To generate the test report run the createTestReport task: .\gradlew.bat createTestReport.

The test report will be generated into the MiniHilProject/build/tests/testReport folder.

In addition to the HTML Report MiniHILTestReport.html the reports content is also generated in XML and json format.

Screenshots

test report overview
test report testsuite

Machine readable report formats

In addition to the HTML report the test results are also generated in XML and json format. You can find the generated files as well as schema files for the XML (MiniHILTestReport.xsd) and json (MiniHILTestReport.schema.json) format in the MiniHilProject/build/tests/testReport folder.

Both, the XML and json format, are based on the same data model and contain the same information.They may contain additional information which are not rendered in the HTML report.

Documenting test cases

In addition to the test results the test report may also contain test documentation. If a cage file contains a comment block for a test case the content of this block is included in the generated HTML report. To format the content Markdown may be used. For a comment block to be associated with the test case the comment block must be placed directly before the test case definition.

The following example shows a test case definition with a comment block. The comment block is associated with the test case TemplateTestCase1 and will be included in the HTML report.

TestSuite TemplateTestSuite:
	TemplateTestCase1
;

/*

## This is a template for a test case

### Description

1. Description of the `test case`
2. Description of the *test italic case*
3. Description of the **test bold case**

[google.com link](http://www.google.com)

```
Some
multiline
code

ABCd
```

---

Some more text.
*/
Step TemplateTestCase1:
	timeout 100 ms
	action
		timekeeper.startTimeMeasurement
		wait 50 ms
	action
		timekeeper.stopTimeMeasurement
;
test report with markdown comment

Adding test case meta information

For easier processing of the test reports by downstream tools such as ALM importers or test management tools the test report may contain meta information for each test case. The meta information is defined using the @MetaInfo annotation. The annotation may be used multiple times to define multiple meta information entries for a test case.

Meta information are arbitrary key-value pairs which may be freely chosen by the user.

Here is an example of a test case definition with meta information followed by the resulting json output:

TestSuite TemplateTestSuite:
	TemplateTestCase1
;

/**
 My test case doc here...
 */
@MetaInfo(key="almId", text="TST-123") 2
@MetaInfo(key="state", text="WIP")      3
Step TemplateTestCase1:
	timeout 100 ms
	action
		timekeeper.startTimeMeasurement
		wait 50 ms
	action
		timekeeper.stopTimeMeasurement
;
{
//[....]
"testSuites" : [ {
    "testCases" : [ {
      "metaInfo" : { 1
        "almId" : "TST-123", 2
        "state" : "WIP" 3
      },
      "comment" : "My test case doc here...",
      "stdout" : "\n[==============================================================================]\n[  BEGIN ] TestSuite TemplateTestSuite\n\n\n[------------------------------------------------------------------------------]\n[ RUN    ] TestCase TemplateTestCase1\n\n[     OK ] TestCase TemplateTestCase1\n\n[------------------------------------------------------------------------------]\n[ PASSED ] TestSuite TemplateTestSuite\n[        ] Executed: 1 Tests\n[        ] Passed  : 1 Tests\n[        ] Skipped : 0 Tests\n[        ] Failed  : 0 Tests\n\n[############################ Test Execution Finished ############################\nTest execution result: successful\n\n  TS No:    0 [ PASSED] : TemplateTestSuite\n\n  Total number of Test Cases: 1\n                      PASSED: 1\n                     SKIPPED: 0\n                      FAILED: 0\n",
      "name" : "TemplateTestCase1",
      "id" : 1,
      "testSuite" : "TemplateTestSuite",
      "result" : "passed",
      "executionTime" : "0.056"
    } ],
    "testTraceDiagram" : "[.....]",
    "name" : "TemplateTestSuite"
  } ]
}